In an InfiniBand™ Architecture network system, channel adapters are devices that generate and consume packets sent and received over a port connected to the InfiniBand™ fabric. A channel adapter port implements queue pairs to provide a virtual interface and communication port for sending and receiving packets. A queue pair has a send queue for sending packets and a receive queue for receiving packets. Communication takes place between a queue pair of a source port and a queue pair of a destination port of respective channel adapters.
Each queue pair is assigned a number referred to as a queue pair number. As each port may use multiple queue pairs, the queue pair number is used for identifying the queue pair in communications between channel adapters. The InfiniBand™ Architecture specification indicates using a twenty-four (24) bit number for the queue pair number. Although not described by the InfiniBand™ Architecture specification, some implementations of the channel adapter may use the queue pair number to also identify other information. For example, a portion of the queue pair number may identify a queue pair context, which is context information related to use of the queue pair. Each queue pair allocated as a communication resource has a queue pair context. Typically, the queue pair context may be stored in a hardware table.
In some channel adapter implementations, one portion of the queue pair number, such as a set of least significant bits, may provide a pointer to the queue pair context. The pointer may be an index into the hardware table storing the queue pair context information. This portion of the queue number is fixed by the value of the index to the queue pair context. Another portion of the queue pair number, such as a set of the most significant bits, may be changeable to form a set of different combinations of queue pair numbers associated with a specific queue pair context. That is, the most significant bits of the queue pair number can be combined with the fixed portion of the index to the queue pair context to provide different queue pair number values for a specific queue pair context. Therefore, the possible combinations of queue pair numbers for a queue pair context is limited to the combination of values that can be generated by the changeable portion of the queue pair number, or the number of bits that can be changed.
Typically, the changeable portions of queue pair numbers are generated from a global counter applicable to all queue pair contexts. The global counter is used to set the most significant bits of the queue pair number assigned to an allocated queue pair. For each allocation of a queue pair number to any queue pair context, the global counter is incremented to represent the next value to be assigned to a queue pair. Using a global counter for all queue pair contexts provides an approach that is adequate under some circumstances. However, this approach also causes problems under other circumstances.
Many queue pairs and queue pair numbers can be allocated and freed in the course of communications in the InfiniBand™ fabric. With the quantity of queue pair numbers and queue pair contexts available for use limited by the channel adapter hardware implementation, queue pair contexts will be re-used. Also, queue pair numbers referencing the queue pair contexts will be re-used. Since the global counter is applicable to all queue pair contexts, the queue pair number provided to a queue pair of a particular queue pair context is random based on the various multitude of queue pair allocations that have occurred and the current value of the global counter. With the randomness of queue pair numbers for a particular queue pair context provided by the global counter, not all the possible combinations of queue pair numbers for a queue pair context may be used. Additionally, certain queue pair numbers may be re-used more of less frequently than others.
Without tracking the use of queue pair numbers, such as using the global counter approach, a queue pair number may be re-used before it should. This causes errors in the communication interfaces of a destination port or source port, such as invalid states and invalid or no responses. For example, a source channel adapter may reference a queue pair number in a destination port that has different information in the queue pair context because the destination port has re-used the queue pair number. The source channel adapter may have not yet released the queue pair number from its previous use. As such, the source channel adapter has a queue pair number that exists in the destination channel adapter but has since changed the information in the queue pair context because it is used for a different communication connection or context.